05-Data Transformation
Kenneth Tay
Oct 8, 2019
Let’s start by loading the dplyr package:
library(dplyr)##
## Attaching package: 'dplyr'## The following objects are masked from 'package:stats':
##
## filter, lag## The following objects are masked from 'package:base':
##
## intersect, setdiff, setequal, unionDid you notice the warning messages? What’s going on there?
It turns out that the dplyr package has a function named filter(), but the stats package, which is automatically loaded when you start an R session, also has a function named filter()! So, if I type the command filter(dataset, ...), how does R know which filter() function to use?
R looks for the function filter() starting with the package that was loaded most recently, and going backwards in time. Since dplyr was the last package loaded, R will assume that we meant dplyr’s version of filter() and use that.
What if I meant the stats version of filter() instead? Is there a way that I can reference it? Yes! We can use “double colon” notation: stats::filter(). (The general syntax for this is packageName::functionName().)
nycflights13
Today we’ll be working with the flights dataset from the nycflights13 package. Let’s load the nycflights13 package and the flights dataset (use install.packages("nycflights13") if you don’t have the package yet):
library(nycflights13)
data(flights)Next, use the ?, str() and View() functions to examine the dataset:
?flights
str(flights)
View(flights)This dataset contains ~336,000 flights that departed from New York City (all 3 airports) in 2013.
Next, just key in the dataset name (i.e. flights):
flights## # A tibble: 336,776 x 19
## year month day dep_time sched_dep_time dep_delay arr_time
## <int> <int> <int> <int> <int> <dbl> <int>
## 1 2013 1 1 517 515 2 830
## 2 2013 1 1 533 529 4 850
## 3 2013 1 1 542 540 2 923
## 4 2013 1 1 544 545 -1 1004
## 5 2013 1 1 554 600 -6 812
## 6 2013 1 1 554 558 -4 740
## 7 2013 1 1 555 600 -5 913
## 8 2013 1 1 557 600 -3 709
## 9 2013 1 1 557 600 -3 838
## 10 2013 1 1 558 600 -2 753
## # … with 336,766 more rows, and 12 more variables: sched_arr_time <int>,
## # arr_delay <dbl>, carrier <chr>, flight <int>, tailnum <chr>,
## # origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
## # minute <dbl>, time_hour <dttm>Did you notice that the output format is different from what we’ve seen before? That’s because previous datasets were in a data structure that we called data frames, while this is in a data structure called a tibble. Don’t worry about the difference: for all intents and purposes, data frames are the same as tibbles.
filter() and logical operations
Since we are here in Stanford, we may only be interested in flights from NYC to SFO. We can use the filter() verb to achieve this:
flights %>% filter(dest == "SFO")## # A tibble: 13,331 x 19
## year month day dep_time sched_dep_time dep_delay arr_time
## <int> <int> <int> <int> <int> <dbl> <int>
## 1 2013 1 1 558 600 -2 923
## 2 2013 1 1 611 600 11 945
## 3 2013 1 1 655 700 -5 1037
## 4 2013 1 1 729 730 -1 1049
## 5 2013 1 1 734 737 -3 1047
## 6 2013 1 1 745 745 0 1135
## 7 2013 1 1 746 746 0 1119
## 8 2013 1 1 803 800 3 1132
## 9 2013 1 1 826 817 9 1145
## 10 2013 1 1 1029 1030 -1 1427
## # … with 13,321 more rows, and 12 more variables: sched_arr_time <int>,
## # arr_delay <dbl>, carrier <chr>, flight <int>, tailnum <chr>,
## # origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
## # minute <dbl>, time_hour <dttm>Note that we used == to test whether dest was equal to "SFO". DO NOT USE =. In programming, = usually means variable assignment.
There are two other international airports near Stanford, San Jose International Airport (“SJC”) and Oakland International Airport (“OAK”). So if we want to analyze flights that people take to get from NYC to Stanford, we should probably include these flights.
flights %>% filter(dest == "SFO" | dest == "SJC" | dest == "OAK")## # A tibble: 13,972 x 19
## year month day dep_time sched_dep_time dep_delay arr_time
## <int> <int> <int> <int> <int> <dbl> <int>
## 1 2013 1 1 558 600 -2 923
## 2 2013 1 1 611 600 11 945
## 3 2013 1 1 655 700 -5 1037
## 4 2013 1 1 729 730 -1 1049
## 5 2013 1 1 734 737 -3 1047
## 6 2013 1 1 745 745 0 1135
## 7 2013 1 1 746 746 0 1119
## 8 2013 1 1 803 800 3 1132
## 9 2013 1 1 826 817 9 1145
## 10 2013 1 1 1029 1030 -1 1427
## # … with 13,962 more rows, and 12 more variables: sched_arr_time <int>,
## # arr_delay <dbl>, carrier <chr>, flight <int>, tailnum <chr>,
## # origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
## # minute <dbl>, time_hour <dttm>The command above filters the dataset and prints it out, but does not retain the output. To keep the extracted dataset for further analysis, we have to assign it to a variable:
Stanford <- flights %>% filter(dest == "SFO" | dest == "SJC" | dest == "OAK")We now have flights from NYC to SFO/SJC/OAK for the entire year. Let’s say that I’m only interested in flights when school is in session (Sep - Jun). Since month is a numeric variable, we could do this:
Stanford %>% filter(month <= 6 | month >= 9)## # A tibble: 11,351 x 19
## year month day dep_time sched_dep_time dep_delay arr_time
## <int> <int> <int> <int> <int> <dbl> <int>
## 1 2013 1 1 558 600 -2 923
## 2 2013 1 1 611 600 11 945
## 3 2013 1 1 655 700 -5 1037
## 4 2013 1 1 729 730 -1 1049
## 5 2013 1 1 734 737 -3 1047
## 6 2013 1 1 745 745 0 1135
## 7 2013 1 1 746 746 0 1119
## 8 2013 1 1 803 800 3 1132
## 9 2013 1 1 826 817 9 1145
## 10 2013 1 1 1029 1030 -1 1427
## # … with 11,341 more rows, and 12 more variables: sched_arr_time <int>,
## # arr_delay <dbl>, carrier <chr>, flight <int>, tailnum <chr>,
## # origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
## # minute <dbl>, time_hour <dttm>or this:
Stanford %>% filter(month != 7 & month != 8)## # A tibble: 11,351 x 19
## year month day dep_time sched_dep_time dep_delay arr_time
## <int> <int> <int> <int> <int> <dbl> <int>
## 1 2013 1 1 558 600 -2 923
## 2 2013 1 1 611 600 11 945
## 3 2013 1 1 655 700 -5 1037
## 4 2013 1 1 729 730 -1 1049
## 5 2013 1 1 734 737 -3 1047
## 6 2013 1 1 745 745 0 1135
## 7 2013 1 1 746 746 0 1119
## 8 2013 1 1 803 800 3 1132
## 9 2013 1 1 826 817 9 1145
## 10 2013 1 1 1029 1030 -1 1427
## # … with 11,341 more rows, and 12 more variables: sched_arr_time <int>,
## # arr_delay <dbl>, carrier <chr>, flight <int>, tailnum <chr>,
## # origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
## # minute <dbl>, time_hour <dttm>select() & rename()
Let’s return to the Stanford dataset (i.e. all flights from NYC to SFO/SJC/OAK). Notice that we have a total of 19 variables. Sometimes our datasets will have hundreds or thousands of variables! Not all of them may be of interest to us. select() allows us to choose a subset of these variables to form a smaller dataset that may be easier to work with.
19 is a pretty small number so we could do our data analysis without dropping any columns, but let’s just try out some commands to get a feel for how select() works.
We can select columns by name: if we just want the year, month and day columns, we can use the following code:
Stanford %>% select(year, month, day)## # A tibble: 13,972 x 3
## year month day
## <int> <int> <int>
## 1 2013 1 1
## 2 2013 1 1
## 3 2013 1 1
## 4 2013 1 1
## 5 2013 1 1
## 6 2013 1 1
## 7 2013 1 1
## 8 2013 1 1
## 9 2013 1 1
## 10 2013 1 1
## # … with 13,962 more rowsIf the columns we want form a contiguous block, then we can use simpler syntax. To select rows from year to arr_delay (inclusive):
Stanford %>% select(year:arr_delay) ## # A tibble: 13,972 x 9
## year month day dep_time sched_dep_time dep_delay arr_time
## <int> <int> <int> <int> <int> <dbl> <int>
## 1 2013 1 1 558 600 -2 923
## 2 2013 1 1 611 600 11 945
## 3 2013 1 1 655 700 -5 1037
## 4 2013 1 1 729 730 -1 1049
## 5 2013 1 1 734 737 -3 1047
## 6 2013 1 1 745 745 0 1135
## 7 2013 1 1 746 746 0 1119
## 8 2013 1 1 803 800 3 1132
## 9 2013 1 1 826 817 9 1145
## 10 2013 1 1 1029 1030 -1 1427
## # … with 13,962 more rows, and 2 more variables: sched_arr_time <int>,
## # arr_delay <dbl>In this example, the year column is superfluous, since all the values are all 2013. The code below drops the year column, keeping the rest:
Stanford %>% select(-year)## # A tibble: 13,972 x 18
## month day dep_time sched_dep_time dep_delay arr_time sched_arr_time
## <int> <int> <int> <int> <dbl> <int> <int>
## 1 1 1 558 600 -2 923 937
## 2 1 1 611 600 11 945 931
## 3 1 1 655 700 -5 1037 1045
## 4 1 1 729 730 -1 1049 1115
## 5 1 1 734 737 -3 1047 1113
## 6 1 1 745 745 0 1135 1125
## 7 1 1 746 746 0 1119 1129
## 8 1 1 803 800 3 1132 1144
## 9 1 1 826 817 9 1145 1158
## 10 1 1 1029 1030 -1 1427 1355
## # … with 13,962 more rows, and 11 more variables: arr_delay <dbl>,
## # carrier <chr>, flight <int>, tailnum <chr>, origin <chr>, dest <chr>,
## # air_time <dbl>, distance <dbl>, hour <dbl>, minute <dbl>,
## # time_hour <dttm>select() can also be used to rearrange the columns. If, for example, I wanted to have the first 3 columns be day, month, year instead of year, month, day:
Stanford %>% select(day, month, year, everything())## # A tibble: 13,972 x 19
## day month year dep_time sched_dep_time dep_delay arr_time
## <int> <int> <int> <int> <int> <dbl> <int>
## 1 1 1 2013 558 600 -2 923
## 2 1 1 2013 611 600 11 945
## 3 1 1 2013 655 700 -5 1037
## 4 1 1 2013 729 730 -1 1049
## 5 1 1 2013 734 737 -3 1047
## 6 1 1 2013 745 745 0 1135
## 7 1 1 2013 746 746 0 1119
## 8 1 1 2013 803 800 3 1132
## 9 1 1 2013 826 817 9 1145
## 10 1 1 2013 1029 1030 -1 1427
## # … with 13,962 more rows, and 12 more variables: sched_arr_time <int>,
## # arr_delay <dbl>, carrier <chr>, flight <int>, tailnum <chr>,
## # origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
## # minute <dbl>, time_hour <dttm>To rename column names, use the rename() function:
Stanford %>% rename(tail_num = tailnum)## # A tibble: 13,972 x 19
## year month day dep_time sched_dep_time dep_delay arr_time
## <int> <int> <int> <int> <int> <dbl> <int>
## 1 2013 1 1 558 600 -2 923
## 2 2013 1 1 611 600 11 945
## 3 2013 1 1 655 700 -5 1037
## 4 2013 1 1 729 730 -1 1049
## 5 2013 1 1 734 737 -3 1047
## 6 2013 1 1 745 745 0 1135
## 7 2013 1 1 746 746 0 1119
## 8 2013 1 1 803 800 3 1132
## 9 2013 1 1 826 817 9 1145
## 10 2013 1 1 1029 1030 -1 1427
## # … with 13,962 more rows, and 12 more variables: sched_arr_time <int>,
## # arr_delay <dbl>, carrier <chr>, flight <int>, tail_num <chr>,
## # origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
## # minute <dbl>, time_hour <dttm>arrange()
Often we get datasets which are not in order, or in an order which we are not interested in. The arrange() function allows us to reorder the rows according to an order we want.
The Stanford dataset looks like it is already ordered by actual departure time. Perhaps I’m most interested in the flights which had the longest departure delay. I could sort the dataset as follows:
Stanford %>% arrange(dep_delay)## # A tibble: 13,972 x 19
## year month day dep_time sched_dep_time dep_delay arr_time
## <int> <int> <int> <int> <int> <dbl> <int>
## 1 2013 12 11 710 730 -20 1039
## 2 2013 11 16 712 730 -18 1025
## 3 2013 9 11 712 730 -18 946
## 4 2013 11 19 713 730 -17 1036
## 5 2013 7 14 1151 1208 -17 1450
## 6 2013 12 10 714 730 -16 1104
## 7 2013 3 29 1050 1106 -16 1359
## 8 2013 4 20 1420 1436 -16 1737
## 9 2013 5 20 719 735 -16 951
## 10 2013 1 23 545 600 -15 948
## # … with 13,962 more rows, and 12 more variables: sched_arr_time <int>,
## # arr_delay <dbl>, carrier <chr>, flight <int>, tailnum <chr>,
## # origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
## # minute <dbl>, time_hour <dttm>Looks like the flights with the shortest delay are at the top instead! To re-order by descending order, use desc():
Stanford %>% arrange(desc(dep_delay))## # A tibble: 13,972 x 19
## year month day dep_time sched_dep_time dep_delay arr_time
## <int> <int> <int> <int> <int> <dbl> <int>
## 1 2013 9 20 1139 1845 1014 1457
## 2 2013 7 7 2123 1030 653 17
## 3 2013 7 7 2059 1030 629 106
## 4 2013 7 6 149 1600 589 456
## 5 2013 7 10 133 1800 453 455
## 6 2013 7 10 2342 1630 432 312
## 7 2013 7 7 2204 1525 399 107
## 8 2013 7 7 2306 1630 396 250
## 9 2013 6 23 1833 1200 393 NA
## 10 2013 7 10 2232 1609 383 138
## # … with 13,962 more rows, and 12 more variables: sched_arr_time <int>,
## # arr_delay <dbl>, carrier <chr>, flight <int>, tailnum <chr>,
## # origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
## # minute <dbl>, time_hour <dttm>(Wow, that’s a really long delay! Almost 17 hours.) To extract just the flights with the top 10 departure delays, we can use the head() function:
Stanford %>%
arrange(desc(dep_delay)) %>%
head(n = 10)## # A tibble: 10 x 19
## year month day dep_time sched_dep_time dep_delay arr_time
## <int> <int> <int> <int> <int> <dbl> <int>
## 1 2013 9 20 1139 1845 1014 1457
## 2 2013 7 7 2123 1030 653 17
## 3 2013 7 7 2059 1030 629 106
## 4 2013 7 6 149 1600 589 456
## 5 2013 7 10 133 1800 453 455
## 6 2013 7 10 2342 1630 432 312
## 7 2013 7 7 2204 1525 399 107
## 8 2013 7 7 2306 1630 396 250
## 9 2013 6 23 1833 1200 393 NA
## 10 2013 7 10 2232 1609 383 138
## # … with 12 more variables: sched_arr_time <int>, arr_delay <dbl>,
## # carrier <chr>, flight <int>, tailnum <chr>, origin <chr>, dest <chr>,
## # air_time <dbl>, distance <dbl>, hour <dbl>, minute <dbl>,
## # time_hour <dttm>arrange() also allows us to filter by more than one column, in that each additional column will be used to break ties in the values of the preceding ones. For example, flights seems to be sorted by year, month, day, and actual departure time. If I wanted to sort by year, month, day and scheduled departure time instead:
Stanford %>% arrange(year, month, day, sched_dep_time)## # A tibble: 13,972 x 19
## year month day dep_time sched_dep_time dep_delay arr_time
## <int> <int> <int> <int> <int> <dbl> <int>
## 1 2013 1 1 558 600 -2 923
## 2 2013 1 1 611 600 11 945
## 3 2013 1 1 655 700 -5 1037
## 4 2013 1 1 729 730 -1 1049
## 5 2013 1 1 734 737 -3 1047
## 6 2013 1 1 745 745 0 1135
## 7 2013 1 1 746 746 0 1119
## 8 2013 1 1 803 800 3 1132
## 9 2013 1 1 826 817 9 1145
## 10 2013 1 1 1029 1030 -1 1427
## # … with 13,962 more rows, and 12 more variables: sched_arr_time <int>,
## # arr_delay <dbl>, carrier <chr>, flight <int>, tailnum <chr>,
## # origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
## # minute <dbl>, time_hour <dttm>mutate()
In this dataset we have both the time the plane spent in the air (air_time) and distance traveled (distance). From these two pieces of information, we can figure out the average speed of the plane for the flight using mutate().
mutate() adds new columns to the end of the dataset, so let’s work with a smaller dataset for now so that we can see the values of our new column.
Stanford_small <- Stanford %>%
select(month, carrier, origin, dest, air_time, distance) %>%
mutate(speed = distance / air_time * 60)
Stanford_small## # A tibble: 13,972 x 7
## month carrier origin dest air_time distance speed
## <int> <chr> <chr> <chr> <dbl> <dbl> <dbl>
## 1 1 UA EWR SFO 361 2565 426.
## 2 1 UA JFK SFO 366 2586 424.
## 3 1 DL JFK SFO 362 2586 429.
## 4 1 VX JFK SFO 356 2586 436.
## 5 1 B6 JFK SFO 350 2586 443.
## 6 1 AA JFK SFO 378 2586 410.
## 7 1 UA EWR SFO 373 2565 413.
## 8 1 UA JFK SFO 369 2586 420.
## 9 1 UA EWR SFO 357 2565 431.
## 10 1 AA JFK SFO 389 2586 399.
## # … with 13,962 more rowsmutate() can be used to create several new variables at once. For example, the following code is valid syntax:
Stanford_small %>% mutate(speed_miles_per_min = air_time / distance,
speed_miles_per_hour = speed_miles_per_min * 60)## # A tibble: 13,972 x 9
## month carrier origin dest air_time distance speed speed_miles_per…
## <int> <chr> <chr> <chr> <dbl> <dbl> <dbl> <dbl>
## 1 1 UA EWR SFO 361 2565 426. 0.141
## 2 1 UA JFK SFO 366 2586 424. 0.142
## 3 1 DL JFK SFO 362 2586 429. 0.140
## 4 1 VX JFK SFO 356 2586 436. 0.138
## 5 1 B6 JFK SFO 350 2586 443. 0.135
## 6 1 AA JFK SFO 378 2586 410. 0.146
## 7 1 UA EWR SFO 373 2565 413. 0.145
## 8 1 UA JFK SFO 369 2586 420. 0.143
## 9 1 UA EWR SFO 357 2565 431. 0.139
## 10 1 AA JFK SFO 389 2586 399. 0.150
## # … with 13,962 more rows, and 1 more variable: speed_miles_per_hour <dbl>If we only want to keep the newly created variables, use transmute() instead of mutate().
A digression: plotting our data
Let’s make use of our data visualization skills to see if there are any trends in air time. First, let’s make a histogram of air_time:
library(ggplot2)
ggplot(data = Stanford_small) +
geom_histogram(aes(x = air_time))## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.## Warning: Removed 162 rows containing non-finite values (stat_bin).
Did you notice the warning message about rows being removed for “containing non-finite values”? If you view the Stanford_small dataset and scroll all the way down, you’ll notice that there are some rows which have NA for air_time. Since we don’t know what the air time is, we won’t be able to compute the speed for those rows.
As a data analyst, NAs are something to watch out for as they could invalidate your analysis. Why are these data missing? Is it completely at random, or is there something going on? For this session, we will just leave them in the dataset.
It seems like the air time of planes might vary depending on the origin and destination, so let’s facet on these 2 variables:
ggplot(data = Stanford_small) +
geom_histogram(aes(x = air_time)) +
facet_grid(origin ~ dest)## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.## Warning: Removed 162 rows containing non-finite values (stat_bin).
We learn 3 things from this plot: (i) there are no flights from La Guardia (LGA) to any of the 3 airports; (ii) there are no flights from Newark (EWR) to SJC/OAK; and (iii) there are very few flights from NYC to SJC/OAK compared to SFO. It’s hard to tell if there are differences in the distributions from this plot. One alternative is to facet in the other direction, then let each facet have its own y-axis:
ggplot(data = Stanford_small) +
geom_histogram(aes(x = air_time)) +
facet_grid(dest ~ origin, scales = "free_y")## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.## Warning: Removed 162 rows containing non-finite values (stat_bin).
summarize()
Instead of looking at plots, we can try to look at summary statistics instead. What was the mean/median air time for flights in our Stanford_small dataset? We can use the summarize() function to help us:
Stanford_small %>% summarize(mean_airtime = mean(air_time))## # A tibble: 1 x 1
## mean_airtime
## <dbl>
## 1 NAStanford_small %>% summarize(median_airtime = median(air_time))## # A tibble: 1 x 1
## median_airtime
## <dbl>
## 1 NAThe NAs are causing us trouble! We need to specify the na.rm = TRUE option to remove NAs from consideration:
Stanford_small %>% summarize(mean_airtime = mean(air_time, na.rm = TRUE))## # A tibble: 1 x 1
## mean_airtime
## <dbl>
## 1 346.Stanford_small %>% summarize(median_airtime = median(air_time, na.rm = TRUE))## # A tibble: 1 x 1
## median_airtime
## <dbl>
## 1 345summarize() gives me a summary of the entire dataset. If I want summaries by group, then I have to use summarize() in conjunction with group_by(). group_by() changes the unit of analysis from the whole dataset to individual groups. The following code groups the dataset by carrier, then computes the summary statistic for each group:
Stanford_small %>%
group_by(carrier) %>%
summarize(mean_airtime = mean(air_time, na.rm = TRUE)) %>%
arrange(desc(mean_airtime))## # A tibble: 5 x 2
## carrier mean_airtime
## <chr> <dbl>
## 1 AA 348.
## 2 VX 348.
## 3 DL 347.
## 4 B6 347.
## 5 UA 344.I can also group by more than one variable. For example, if I wanted to count the number of flights for each carrier in each month, I could use the following code:
Stanford_small %>%
group_by(month, carrier) %>%
summarize(count = n())## # A tibble: 60 x 3
## # Groups: month [12]
## month carrier count
## <int> <chr> <int>
## 1 1 AA 120
## 2 1 B6 121
## 3 1 DL 142
## 4 1 UA 422
## 5 1 VX 124
## 6 2 AA 108
## 7 2 B6 106
## 8 2 DL 127
## 9 2 UA 378
## 10 2 VX 104
## # … with 50 more rowsWe can even “pipe” the dataset to ggplot() to plot the data!
Stanford_small %>%
group_by(month, carrier) %>%
summarize(count = n()) %>%
ggplot(mapping = aes(x = month, y = count, col = carrier)) +
geom_line() +
geom_point() +
scale_x_continuous(breaks = 1:12)
Optional material
The %in% operator
Recall that we used the following line of code to extract flights that landed in SFO, SJC or OAK:
Stanford <- flights %>% filter(dest == "SFO" | dest == "SJC" | dest == "OAK")We can use the %in% operator to make the code more succinct:
flights %>% filter(dest %in% c("SFO", "SJC", "OAK"))## # A tibble: 13,972 x 19
## year month day dep_time sched_dep_time dep_delay arr_time
## <int> <int> <int> <int> <int> <dbl> <int>
## 1 2013 1 1 558 600 -2 923
## 2 2013 1 1 611 600 11 945
## 3 2013 1 1 655 700 -5 1037
## 4 2013 1 1 729 730 -1 1049
## 5 2013 1 1 734 737 -3 1047
## 6 2013 1 1 745 745 0 1135
## 7 2013 1 1 746 746 0 1119
## 8 2013 1 1 803 800 3 1132
## 9 2013 1 1 826 817 9 1145
## 10 2013 1 1 1029 1030 -1 1427
## # … with 13,962 more rows, and 12 more variables: sched_arr_time <int>,
## # arr_delay <dbl>, carrier <chr>, flight <int>, tailnum <chr>,
## # origin <chr>, dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
## # minute <dbl>, time_hour <dttm>The %in% operator is very useful, especially we are checking if dest belongs to a long list of airports.
Joy plots
Let’s remove the rows with air_time being NA:
Stanford_small <- Stanford_small %>%
filter(!is.na(air_time))One theory we might have is that different carriers have different air times. Let’s do a facet on carrier:
ggplot(data = Stanford_small) +
geom_histogram(aes(x = air_time)) +
facet_grid(carrier ~ .)## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
The first thing we notice is that UA has many more flights than the other carriers. Because all 5 histograms have the same y-axis, this causes the other histograms to be obscured. To allow each histogram to have its own y-axis, we can add a scales argument to facet_grid():
ggplot(data = Stanford_small) +
geom_histogram(mapping = aes(x = air_time)) +
facet_grid(carrier ~ ., scales = "free_y")## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
As you can see, the histograms have very similar shapes, suggesting that the air times of various carriers is roughly the same. The one thing that we might notice is are the tails on the right.
A plot that is increasing in popularity for plotting multiple histograms or density plots is the joy plot. The plot looks like a series of overlapping mountain ranges which can be compared against each other more easily than the histograms. The code below produces a joy plot:
library(ggridges)##
## Attaching package: 'ggridges'## The following object is masked from 'package:ggplot2':
##
## scale_discrete_manualggplot(data = Stanford_small, aes(x = air_time, y = carrier)) +
geom_density_ridges(scale = 5)## Picking joint bandwidth of 3.24
(Play around with the scale parameter and see what happens.)
Session info
sessionInfo()## R version 3.6.1 (2019-07-05)
## Platform: x86_64-apple-darwin15.6.0 (64-bit)
## Running under: macOS Mojave 10.14.5
##
## Matrix products: default
## BLAS: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRblas.0.dylib
## LAPACK: /Library/Frameworks/R.framework/Versions/3.6/Resources/lib/libRlapack.dylib
##
## locale:
## [1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
##
## attached base packages:
## [1] stats graphics grDevices utils datasets methods base
##
## other attached packages:
## [1] ggridges_0.5.1 ggplot2_3.2.1 nycflights13_1.0.0
## [4] dplyr_0.8.3 knitr_1.24
##
## loaded via a namespace (and not attached):
## [1] Rcpp_1.0.2 plyr_1.8.4 pillar_1.4.2 compiler_3.6.1
## [5] tools_3.6.1 zeallot_0.1.0 digest_0.6.20 evaluate_0.14
## [9] tibble_2.1.3 gtable_0.3.0 pkgconfig_2.0.2 rlang_0.4.0
## [13] cli_1.1.0 yaml_2.2.0 xfun_0.9 withr_2.1.2
## [17] stringr_1.4.0 vctrs_0.2.0 grid_3.6.1 tidyselect_0.2.5
## [21] glue_1.3.1 R6_2.4.0 fansi_0.4.0 rmarkdown_1.15
## [25] reshape2_1.4.3 purrr_0.3.2 magrittr_1.5 backports_1.1.4
## [29] scales_1.0.0 htmltools_0.3.6 assertthat_0.2.1 colorspace_1.4-1
## [33] labeling_0.3 utf8_1.1.4 stringi_1.4.3 lazyeval_0.2.2
## [37] munsell_0.5.0 crayon_1.3.4